Interactive rf system testing system and method

ABSTRACT

A system and method for evaluating the interactivity of RF devices in a virtual RF environment having selective virtual spectrum users remotely controlled by a web browser, where the virtual spectrum users have selectable interactivity parameters, and the virtual RF spectrum can be selectively changed, and the performance of the virtual spectrum users is evaluated and assigned scores as a function of the evaluation to determine which virtual spectrum user receives the highest amount of points.

This application is a continuation in part to U.S. patent applicationSer. No. 14/011,278 filed Aug. 27, 2013, which claims priority to U.S.Pat. No. 8,521,092 filed May 26, 2010, which claims the priority of U.S.Provisional Pat. App. Ser. No. 61/217,001 filed on May 27, 2009, thedisclosures of which are hereby incorporated by reference. Thisapplication also claims priority to U.S. Provisional Pat. App. Ser. No.61/783,199 filed Mar. 14, 2013.

The present disclosure relates to the field of behavior evaluation ofradio frequency (“RF”) systems. More specifically, this disclosuredescribes a system and method for providing a laboratory-based, fieldrealistic, virtual RF environment where RF systems (communications,radar, jammers, etc.) produced by unassociated third parties canparticipate in an interactive gaming environment to evaluate behavior.

BACKGROUND

Radio spectrum is scarce and the Federal Communications Commission(“FCC”), Department of Defense (“DoD”) and other international spectrummanagement organizations are constantly looking for ways to moreefficiently utilize this limited spectrum. Demand for spectrum iscontinuing to rise due to the explosive growth of data, voice,messaging, and video applications. One solution to meeting the need forimproved spectral efficiency as measured by bits/Hz/user is adaptiveradios (also referred to as dynamic spectrum access (DSA) or cognitiveradios (CR)). Adaptive radios can change their transmissioncharacteristics to maximize transmission capacity and coverage whileconserving spectral usage. Because CR represents a rich technicalresearch area, as well as a potentially significant commercial andmilitary products market opportunity, large numbers of research anddevelopment entities from academic, commercial and governmentorganizations are participating in activities towards producing CRdevices and systems.

One of the challenges of deploying adaptive radio technology is that itcannot be fielded without comprehensive behavior evaluation, and itcannot be evaluated in a densely populated, live environment for fear ofpotentially interfering with existing spectrum users (primary users).Field evaluation is preferable to lab evaluation but requires arealistic environment where it can be verified that the System underTest (SUT) will not interfere with primary users or other spectrumusers. Laboratory evaluation is more cost and schedule effective,repeatable, controllable and observable, but generally lacks in realism,especially with respect to RF environmental considerations including theinteractive effects of other devices/systems operating in the RFenvironment. These interactive effects can be between devices/systemsthat are adaptive/cognitive and operate in different spectrum bands, butdynamically change their frequencies in response to spectrum conditions.Addressing this phenomena has not been a part of prior art testingapproaches.

There is an established and growing need to comprehensively evaluatebehavior of these new adaptive devices/systems in known and postulatedenvironments which include other representative RF systems to establishbehavior characteristics. Traditional evaluation methods areincreasingly stressed by the proliferation and diversity of thedevices/systems and operating environments. Historically, device/systemevaluation has fallen into two broad categories, field evaluation andlaboratory simulation/evaluation. Field evaluation as illustrated inFIG. 1 involves placing some number of devices in a realistic fieldenvironment and exercising them to evaluate performance againstspecified functionality. Full-featured field evaluations place thewireless transceivers in a field scenario containing some representativeRF environment where they will be operated while evaluation data iscollected. These sorts of evaluations are often expensive and complex toorchestrate, and can lack flexibility since mixes of test transceivernumbers/types/locations, incumbent RF user numbers/types/locations andRF propagation conditions cannot be systematically varied to collectcomprehensive data. FIG. 1 schematically depicts a typical fieldevaluation equipment setup. Wireless Transceiver Units Under Test (UUT)100 operate in some RF environment 110. The RF emissions are subject tothe noise, path loss, multipath transmission and interferers found inthe local RF environment 110. Test instrumentation 120 is established tomeasure the performance of the UUT and other primary users (PU) of theRF environment. In order to accomplish a field evaluation of thisvariety, the UUT 100 must be physically located in the evaluation RFenvironment 110, and test instrumentation 120 must be constructed. Inorder to vary the numbers/types/locations of UUT and PU, physical unitsmust be acquired and placed in the RF environment. In order to vary theRF environment, different field venues must be available. Additionally,test instrumentation must be provided and adapted for each UUT/PU/testenvironment scenario where testing is to be accomplished.

Many factors must be considered when selecting and configuring the fieldevaluation area including the specific type and host platform for theSUT, the characteristics and quantity of other RF devices andinterferers in the environment, and environmental factors that affectthe radio propagation including terrain and morphology. Field evaluationmethods have been viewed as the most realistic, but many growingchallenges limit their ability to be compelling. These challengesinclude:

-   -   Difficulty and complexity in evaluating high platform dynamic        systems    -   More devices/systems to evaluate    -   More functionality & complexity to exercise including        adaptive/cognitive behavior    -   Evaluation ranges require a broad set of realistic physical        layouts    -   Requirements to emulate location-specific RF environments        including propagation and interferers    -   Requirements for conditions not realizable on evaluation ranges        including prohibition by FCC rules    -   RF environment control difficult due to encroachment of        commercial RF sources.

All of the above lead to increased costs, longer schedules, morerequirements on field evaluation assets and ranges, and potentiallylower confidence in results.

For adaptive RF systems, field evaluation is not practical. Laboratoryevaluation methods are generally more cost and schedule effective, aremore controllable and observable, but generally are lacking in realism,especially with respect to RF environmental considerations includingother RF sources.

There exist many variations of lab evaluation approaches, but they canbe generally bounded by “RF Path Simulator” and “Software Modeling”variants. The RF Path Simulator approach shown in FIG. 2, whichinterconnects RF systems/devices with conventional laboratory testequipment such as signal/noise generators, is only applicable to simpleRF environments, small numbers of devices/systems under test with simpleantenna systems, and small number of primary users/interferers.Lab-based evaluation using cable-based interconnection for RF emissionsof UUT and the RF environment is a prior art approach to testing toovercome the challenges of placing and monitoring devices in the fieldenvironment. FIG. 2 depicts a typical lab-based equipment setup. As infield evaluations, Wireless Transceiver Units Under Test (UUT) 100 areacquired and instrumented with Test Instrumentation 120. Instead of theRF environment being that found in the field, RF test equipment such assignal generators are used to produce Interferers 210, Noise Generators220, and Path Simulators 200 to simulate path loss and multipath in anRF channel. RF Interconnection 230 is accomplished using RF cables suchas coaxial cables. This test set up approach reduces some of thecomplexities of field evaluation, but introduces new concerns over RFenvironment realism. Further, it still requires the physicalintroduction of new UUT and RF test equipment into the configuration forcomprehensive transceiver configuration and RF environment results.

Traditional methods that use software modeling approaches as shown inFIG. 3 have historically made simplifications about the physicalenvironment/radio propagation effects, and generally cannot support anyhardware in the loop (HITL) test cases. Their validity is thereforelimited to a narrow group of test cases and not well suited to theadaptive RF system evaluation problem. A variation on RF cable-connectedlab testing has become more prevalent and straightforward as wirelesstransceiver devices have tended towards digital waveforms and digitalhardware or software implementation. FIG. 3 depicts a typical frameworkfor modern wireless communications devices as defined by the prior artOSI model. Here, different functions in the Wireless Transceiver 100 areallocated to layers in the functional stack 300. The physical layer instack 300 is where the waveform-related functionality is contained. Thephysical layer can be segregated into a digital implementation portion310 and an analog portion 320. Typical functions in the digital transmitportion 310 are waveform generation 330 and digital to analog conversion340. Typical functions found in the analog portion 320 are baseband toRF conversion 350. Other digital processing functions associated withnon-physical layers (2 through 7) are performed through digital dataprocessing blocks 360.

A laboratory-based evaluation approach that combines the advantages oftrue RF path/environment emulation and HITL, but implemented in thedigital domain under software control, has the potential to deliver theadvantages of the different lab methods with the realism of fieldtesting. The test platform disclosed in commonly owned U.S. Pat. No.8,521,092, titled “Wireless Transceiver Test Bed System and Method”,which is hereby incorporated by reference, follows this approach. Thepresent disclosure adds improvements directed to a system and method forproviding a laboratory-based, field realistic, virtual RF environmentwhere RF systems (communications, radar, jammers, etc.) produced byunassociated third parties can participate in an interactive gamingenvironment to evaluate behavior. This facet of the test bed problem isfurther described below.

FIG. 4 of U.S. Pat. No. 8,521,092, titled “Wireless Transceiver Test BedSystem and Method” is included as FIG. 4 in the current disclosure todescribe the operation of one embodiment. FIG. 4 illustrates the virtualwireless channel (VWC) 400 and test instrumentation plane (TIP) andmetadata manager 410. The TIP may also be embodied as a database. TheUUT physical layer digital portion is connected to the VWC 400 viainterconnections 420, as is the TIP via 425. The VWC function is toprovide a realistic wireless channel model including noise,interference, UUT signal path loss and UUT signal multipathtransmission. The VWC 400 can be configured with a selectable number ofvirtual spectrum users (VSU) and other interferers to accuratelysimulate the RF environment that might be encountered in different partsof the world. The VSU may have selectable interactivity parameters,including transmission parameters and kinetics, or physicalcharacteristics. For example, transmission parameters may includefrequency, bandwidth, power, modulation. Physical characteristics, orkinetics, may include location, speed, direction of motion, and antennaparameters including type, elevation gain, azimuth gain, phase,polarization and orientation. The VSU can be selected to be atransmitter only, a receiver only or a transceiver. The VSU can beselected to be a communication device, a sensor such as a radar, anavigation device, or a jammer and can be the same type or differentthan the UUT. The VWC also allows for selecting transmission parametersand physical characteristics of the physical UUT.

A key feature of the VWC is that it accepts and passes analog RF ordigitized RF to and from the UUT. In this way, the full effects of thewireless channel can be included in the simulation. The TIP 410 acts asa control mechanism to orchestrate the sequencing of the test bedsimulation, and to collect instrumentation data at the RF and other OSIlayers of the UUT. A key part of the TIP is the metadata manager.Metadata is defined as data that must be passed between the VWC and theUUT to allow real time parameters to be modeled and analyzed. As anexample, metadata can include the relative locations of the UUT and VSUin a geographic region. As the simulation progresses, the delaycharacteristics of the multipath and relative time of arrival of thesignals at each node can be accurately modeled.

Perhaps the most challenging part of adaptive RF system behaviorevaluation is addressing the interaction between RF systems (includingadaptive RF systems) in the field. An anticipatable adaptive RF systembehavior pattern (“system 1”) may be that it adapts in response toanother RF system (“system 2”) in the field (like changing RF frequencyof operation), which causes system 1 to adapt (like lowering itstransmission pattern), which causes system 2 to adapt (by changing RFfrequency back to its original center frequency), and so on. Theseconditions are not producible in the field or laboratory today, in part,because many of the adaptive RF systems that will be in the field in thefuture do not exist today in either a “test equipment” form or“prototype form” to facilitate behavior evaluation. In fact, many futureadaptive devices are only available as laboratory R&D models inuniversity, commercial and government R&D facilities.

Based on a review of the available RF system test beds that exist inindustry and academia (including those referenced in U.S. Pat. No.8,521,092), a wireless transceiver test bed approach, capable ofallowing unassociated third parties to participate in interactivespectrum gaming environments to evaluate behavior is not known.

The present disclosure utilizes emerging technologies and trends in theareas of computer networking, digital signal processing, wireless devicedesign, wideband networks, computer and software architecture/capabilityand software-based modeling to provide a means to address theseshortcomings. Specific technology innovations that contribute to variousaspects of the present disclosure include:

-   -   digital signal processing power and available algorithms and        models    -   ability to digitize RF with high fidelity    -   emerging software defined radio (SDR) software architectures,        such as SCA (Software Communications Architecture)    -   emerging commercial off-the-shelf digital radio and SDR        components (hardware and software)    -   ever increasing broadband connectivity between distributed sites    -   comprehensive and advanced RF propagation models    -   RF emitter models being built in software    -   proliferation of radio functionality being digital and        implemented in software with discrete events (bits, bursts,        frames, etc.).    -   standardization of baseband digitized interfaces to SDRs (such        as the VITA-49 Radio Transport Protocol).

The present disclosure is not limited to adaptive wireless devices inthe application area of communications, but broadly applies to allwireless devices and networks including receive only, transmit only anddiverse applications such as sensing, radar, and jamming. Further, it isnot limited to behavior evaluation of adaptive RF systems and could alsobe used to evaluate conventional RF systems.

In summary, a large number of organizations are involved in thedevelopment of adaptive RF systems including industry, academia, andgovernment. Methods, tools, and metrics to collaboratively andcomparatively judge the behavior of these systems (either individuallyor interactively) do not exist. Progress in maturing the designs forcognitive RF systems, understanding their performance, and introducingthem into the field are hampered by the lack of behavior evaluationcapabilities. The disclosed system provides a means to enable thebehavior evaluation in a cost effective and engaging way.

SUMMARY OF DISCLOSURE

The disclosed system creates an interactive virtual RF environment whereRF devices operate and/or compete with other RF devices or theenvironment. A useful analogy is a video game where opposing playerscreate real time strategies to battle each other or thecomputer-controlled enemy; or where many players participate in amassively multi-player online role-playing game (for example, thecommercial game Warcraft).

The disclosed system could be used in a fashion comparable to the DARPArobot challenge. DARPA's goals for robot challenge are similar to theuse goals for the disclosed system which are: facilitating thedevelopment of advanced robotic capabilities; making robot technologymore accessible, and creating a widely available, validated, affordable,community-supported, and enhanced virtual test environment (DARPAequates this last goal to the development of SPICE (Simulation Programwith Integrated Circuit Emphasis) for integrated circuits). A keyattribute of the DARPA robot challenge virtual test environment iseliminating the need for physical prototyping in the evaluation ofhardware and software designs. Similarly, a key attribute of theproposed system is to allow developmental cognitive RF system designs tobe evaluated without creating a full RF hardware suite by implementingthe testing at digital baseband (“digitized RF”, where the RF hardwarecan be modeled if desired). The disclosed system also make available RFsystem building blocks to challengers to build new RF systems.

In one aspect, the present disclosure is a method of evaluatinginteracting RF devices, including receiving selected interactivityparameters for a first virtual spectrum user, receiving selectedinteractivity parameters for a second virtual spectrum user, providing avirtual RF environment for the first and second remote spectrum users,evaluating the performance of the first and second virtual spectrumusers in the virtual RF environment, assigning a score to the firstvirtual spectrum user as a function of the evaluated performance of thefirst virtual spectrum user, assigning a score to the second virtualspectrum user as a function of the evaluated performance of the secondvirtual spectrum user, and identifying whether the first or secondvirtual spectrum user was assigned the higher score.

In another aspect, the present disclosure is a system for evaluatinginteracting RF including a first virtual spectrum user having firstselectable interactivity parameters, a second virtual spectrum userhaving second selectable interactivity parameters, a first remote userfor controlling the first virtual spectrum user through a first webbrowser, a second remote user for controlling the second virtualspectrum user through a second web browser, a real-time modelerprocessor responsive to the first and second virtual spectrum users inreal time to track the physical location of the first and second virtualspectrum users, a virtual spectrum processor responsive to the real-timemodeler processor to emulate a virtual RF environment in which the firstand second virtual spectrum users operate, a database of the first andsecond selectable interactivity parameters for the first and secondvirtual spectrum users in communication with the first and second remoteusers, and an evaluation processor for evaluating the performance of thefirst and second virtual spectrum in the virtual RF environment andassigning a score to the first and second virtual spectrum users as afunction of the evaluation.

The disclosed system incorporates functionality from two sources. Thefirst is the test platform disclosed in commonly owned U.S. Pat. No.8,521,092 titled “Wireless Transceiver Test Bed System and Method”. Inthis disclosed system, third parties can attach “spectrum users” atdigital baseband through an open interface and interact with the virtualRF environment including accurate propagation (terrain-appropriate pathloss, fading, multipath, Doppler, and delay), host platform motion,antenna patterns, environmental interferers, other spectrum users, etc.Second, the disclosed system architecture can be implemented with IPinterconnections, which facilitates third party/multi-platformconfigurations, and also allows processing and GUI functions to beseparated and operated via a web browser or similar means. With thesefeatures, the functionality of remote users operating their RF systemsthrough their web browsers while interacting with other RF systems in avirtual RF environment can be realized.

With reference to FIG. 5, in one embodiment, the disclosed system allowsvirtual spectrum users (VSUs), attributable to third parties, to existin the Software-based Emulation Platform 500. These VSUs can be providedby third parties, and/or controlled/monitored by third parties. Fordisclosure clarity, we will refer to these VSUs as being ContestantVSUs, and the operator of the Software-based Emulation Platform as theSpectrum Master. FIG. 5 shows these Contestant VSUs 530 in the contextof the Software-based Emulation Platform. Their interconnection to theother functions in the Software-based Environment Emulator aresubstantially similar to those for non-Contestant VSUs 540. ContestantVSUs 530 can be controlled and/or monitored remotely through theinternet 520 using Contestant Clients 510. For disclosure clarity, wewill refer to a time session where VSUs interact with the VirtualWireless Channel 550 and Instrumentation data 560 is collected as“Spectrum Wars” (“game duration or period over which VSU behavior isbeing evaluated).

The following is an example of a Spectrum Wars game scenario toillustrate how strategies and scoring can occur. Some number of partiesparticipate in the game. The first participant is the Spectrum Masterwho establishes the conditions for the game including the geographicarea, the spectrum availability rules (“policy”) andnumber/identity/roles of the players. The other players compete forpoints during the game (“Contestants”). Contestants may provide acommunications link/network/jammers/radars/etc. as VSUs 530, configuredin the context of the spectrum policy and the selected geographic area.He also selects a vehicular host platform and antenna system.

The contestants gain points as the session progresses by performingtheir intended function (i.e. communicating), and/or not interferingwith other contestants. As the game progresses, all players havedisplays showing selected geographic and spectral activity, informationabout the behavior of their VSU, and scoring information. The SpectrumMaster has access to all information (it is also recorded for postanalysis). Each contestant has access to other contestant informationthrough the emulated spectrum only, just as in the real-world.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a simplified block diagram illustrating the components in atypical prior art field-based testing configuration for an RF device orsystem.

FIG. 2 is a simplified block diagram illustrating the components in atypical prior art RF interconnected laboratory-based testingconfiguration for an RF device or system.

FIG. 3 is a simplified block diagram illustrating the components in atypical prior art software model-based testing configuration for an RFdevice or system.

FIG. 4 is a simplified block diagram illustrating the placement andfunction of a virtual wireless channel in a laboratory-based testingconfiguration for use in the present disclosure.

FIG. 5 is a is a simplified block diagram illustrating one embodiment ofthe present disclosure showing the placement and function of aContestant VSUs and Contestant Clients interconnected over the internet.

FIG. 6 illustrates one embodiment of a Software-based EvaluationPlatform with components to support Spectrum Wars.

FIG. 7 illustrates one embodiment of a Contestant Client operationalflow.

FIG. 8 illustrates one embodiment of the functional flow of a VSU tosupport streaming data transfer.

DETAILED DESCRIPTION

FIG. 6 illustrates one embodiment of a distributed system architectureused to host “Spectrum Wars”. In its simplest form, it is comprised oftwo distinct components. The first component is a locally hosted ServerFarm 600 which provides much of the computational power for the game.The second component includes Contestants Clients 610 that interfacewith the Server Farm 600 via the internet.

Furthermore, the server farm is made up of the DYnamic SpectrumEnvironment emulator (DYSE) 620, the DYSE Real-Time Modeler 630 and anynumber of VSU hosts 640. DYSE 620 provides an interface for the SpectrumMaster via the Spectrum Master GUI 650, and also serves as the RFemulation host via the Virtual Wireless Channel 660. In this embodiment,DYSE 620 functionality can be distributed between two compute hosts, theSpectrum Master GUI 650 and the VWC 660. The Spectrum Master GUI 650 maybe any specially programmed general purpose computer, such as a WindowsPC. The Spectrum Master GUI may be specifically programmed to specifythe entire RF operating environment. This may include specifying thecontest location and time, the Contestant VSUs, the primary users, thejammers, etc., the contest duration, the rules of the game and scoringmethod. In this embodiment, the DYSE virtual wireless channel 660 may beimplemented by graphics processor units (GPUs) (“GPU Engine”). The DYSEGPU Engine may be a high powered, multi GPU based Linux PC whichemulates the propagation path between all RF entities in the contest inreal time. FIG. 9 illustrates one embodiment of a DYSE GPU Engine.

With reference to FIG. 6, the VSU Hosts 640 may be implemented asvirtual machines that reside on a separate server class PC (VSU Server 1. . . VSU Server N 670). In one embodiment, there is one VSU Host percontestant and per primary user, jammer, etc. The number of allowablecontestants may scale with the number of server class PCs in theSpectrum Wars system architecture. Although VSU Hosts are associatedwith contestants, in one embodiment, it is desirable to host themlocally in the Spectrum Wars system architecture due to the bandwidthrequirements of VSUs, which cannot be guaranteed over the bandwidthlimited internet.

The Contestant Clients 610 can communicate remotely via for example, aweb browser into their respective VSU hosts 640. They can be thought ofas “remote desktop” clients. Using this approach, complete control ofVSUs by the contestants can be achieved without requiring the VSUs toreside locally on the Contestant Client.

In a typical operational scenario, the Spectrum Master sets up the game.For each contestant in the game, the Spectrum Master 650 signals the VSUServer(s) 670 to instantiate a VSU Host 640 and then initiates the RFemulation on the DYSE GPU Engine 660. During the RF emulation, each VSUHost 640 and thus each contestant 610 streams IF/Digital data 665 to theDYSE GPU Engine 660. While this is happening, the Contestant Client 610has full visibility into, and control of, his respective VSU 690 and canalter (through prior VSU programming or in real time) signal strength,wave form, frequency, etc. to try and score points.

In addition to the components described above, a VSU Building BlocksLibrary 680, which may reside in a database on a server farm, can beused by contestants to construct their VSU 690. The VSU Building BlocksLibrary 680 is comprised of VSU components that can be linked togetherto fully define a VSU.

In one embodiment, a Real Time Modeler (RTM) 630 can be incorporated onanother distributed host in the Spectrum Wars architecture to allowContestants the additional flexibility of changing their VSU physicallocation during the emulation. For example, the VSU Hosts 640 cancommunicate their new location, as specified by the Contestant Client610, to the RTM 630. The RTM 630 in turn, re-computes the propagationpath coefficients associated with all transmit and receive pathsaffected by this change in position, and passes them to the DYSE ComputeEngine 660 to be used in real time path loss calculations. In oneembodiment, the Spectrum Master creates 650 a KML stream 655representative of the entire scenario. The stream is sent to eachContestant Client 610 and is fed to Google Earth via a resident customapplication. Google Earth will display a global map of the scenario.FIG. 10 illustrates one embodiment of a real time modeler.

Before the game begins the Spectrum Master decides which contestantswill participate and notifies them for (for example, via email) of theirselection. It also informs them of other necessary game relatedadministrative information such as their location (IP) and (remotedesktop login) credentials.

FIG. 7 is block a diagram of one embodiment of a Contestant Client 700.Upon receiving credentials, the Contestant may log into his respectiveVSU host 710 and can either assemble his VSU from the VSU BuildingBlocks Library 720 or select a previously built VSU. VSUs can be writtenin either an interpreted programming language (Matlab, Python) or acompiled programming language (C/C++). If the language is compiled thanit must be built and assembled directly on the VSU Host 730. In oneembodiment, it cannot be built locally as the Contestant Client Host CPUmay not match that of the VSU Host Server, which is where it will beexecuted. Interpreted languages have no dependency on CPU architectureso they can be constructed anywhere, although they too must be executedon the VSU Host. Once the VSU has been created or selected, it is thenloaded onto the VSU host and instantiated 740. Next, signaling to andfrom DYSE is established 750 and the processes begin execution. Duringexecution, the contestant, who now controls aspects of his VSU through aweb browser interface 760, may relocate the VSU. If he chooses to changeposition, then the new location is passed to the RTM 770. In oneembodiment, during execution, IF/Digital data is passed to/from DYSEto/from the VSU 780. Execution continues until the game ends.

The shaded blocks in FIG. 7 represent one embodiment of a method used togive each Contestant Client full visibility into the entire scenario. Inaddition to what is described above, the Spectrum Master creates a KMLstream representative of the entire scenario[780. The stream may be sentto each Contestant Client and is fed to Google Earth via a residentcustom application. Google Earth will display a global map of thescenario 790.

FIG. 8 shows a functional block diagram of one embodiment of a streamingVSU 800. In this embodiment, VSUs can either be interpreted (Matlab) 810or compiled (C/C++) 820. In either case all communications to DYSE areprovided through a VSU Gateway 830 that may reside on the VSU Host 800.This effectively abstracts the details of the low level communicationschannel from the VSU developer. The API for communications with the VSUGateway 830 is simple and well documented. Also shown in FIG. 8 is aflow chart identifying one embodiment for both transmit 840 and receiveVSUs 850. Receiver VSUs 850 may receive sample data by sending requeststo DYSE 851. They then may wait until DYSE responds with a batch ofsample data and then process it. Processing usually entails executing analgorithm on the data and then visually displaying the processingresults 852. This may lead to altering the processing on subsequentbatches of data. Transmitter VSUs operate in reverse fashion. Initiallya transmitting VSU may receive a data request [841] from the gateway andthen it generates samples as prescribed by an algorithm. It then stuffsthose samples into a batch data message response and forwards thatmessage to the gateway[842. The gateway sends the data on to DYSE.

In one embodiment, the present disclosure can be used in the context ofa Spectrum Wars game scenario, For example, four parties participate inthe game. The first participant is the Spectrum Master who establishesthe conditions for the game including the geographic area, the spectrumavailability rules (“policy”) and number/identity/roles of the players.The other three players compete for points during the game(“Contestants”). The first Contestant provides a communicationslink/network as VSUs, configured in the context of the spectrum policyand the selected geographic area. He also selects a vehicular hostplatform and antenna system. The second Contestant provides a jammer asa VSU. He also configures the VSU and selects a vehicular host platformand antenna system. The third Contestant provides a set of “primaryusers” as VSUs who are stationary in the geographic area. He alsoconfigures the VSU and selects locations for his primary users (inaccordance with Spectrum Master guidance).

The contestants gain points as the session progresses as follows:

-   -   the communications network gains points by measuring throughput        on his link(s), and loses points when he interferes with primary        users    -   The jammer gains points when he reduces throughput of the        communications network link(s), and loses points when he        interferes with primary users    -   the primary users gain points when they detect that the        communications or jammer contestants are creating interference        in the primary user systems.

As the game progresses, all players have displays showing selectedgeographic and spectral activity, information about the behavior oftheir VSU, and scoring information. The Spectrum Master has access toall information (it is also recorded for post analysis). Each contestanthas access to other contestant information through the emulated spectrumonly, just as in the real-world. Contestants are free to execute theirown algorithms and signal processing techniques in the VSU as needed fortheir own application areas (spectral sensing, DF, geolocation,exploitation, jamming, spoofing, etc.). A handicapping scheme is used tonormalize point scoring and deductions to arrive at a game winner.

Another example of a Spectrum War example is an embodiment with threeparties participate in the game. The first participant is again theSpectrum Master who establishes the conditions for the game includingthe geographic area, the spectrum availability rules (“policy”) andnumber/identity/roles of the players. The other two players compete forpoints during the game (“Contestants”). These two Contestants providescommunications link/network as VSUs, configured in the context of thespectrum policy and the selected geographic area. They also select avehicular host platform and antenna system.

The contestants gain points as the session progresses as follows:

-   -   the communications network gains points by measuring throughput        on his link(s), and loses points when he interferes with primary        users installed in the scenario by the Spectrum Master    -   the contestant with the most points at the end wins    -   this game pits two like VSU against one another vs. the prior        game which included three different types of VSUs as        contestants.

As the game progresses, the players again have displays showing selectedgeographic and spectral activity, information about the behavior oftheir VSU, and scoring information. The Spectrum Master has access toall information (it is also recorded for post analysis). Each contestantagain has access to other contestant information through the emulatedspectrum only, just as in the real-world.

It may be emphasized that the above-described embodiments, particularlyany “preferred” embodiments, are merely possible examples ofimplementations, merely set forth for a clear understanding of theprinciples of the disclosure. Many variations and modifications may bemade to the above-described embodiments of the disclosure withoutdeparting substantially from the spirit and principles of thedisclosure. All such modifications and variations are intended to beincluded herein within the scope of this disclosure and the presentdisclosure and protected by the following claims Embodiments of thesubject matter and the functional operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer program products, i.e., one or more modules of computer programinstructions encoded on a tangible program carrier for execution by, orto control the operation of, data processing apparatus. The tangibleprogram carrier can be a computer readable medium. The computer readablemedium can be a machine-readable storage device, a machine-readablestorage substrate, a memory device, a composition of matter affecting amachine-readable propagated signal, or a combination of one or more ofthem.

The term “circuitry” encompasses all apparatus, devices, and machinesfor processing data, including by way of example a programmableprocessor, a computer, or multiple processors or computers. Thecircuitry can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program does notnecessarily correspond to a file in a file system. A program can bestored in a portion of a file that holds other programs or data (e.g.,one or more scripts stored in a markup language document), in a singlefile dedicated to the program in question, or in multiple coordinatedfiles (e.g., files that store one or more modules, sub programs, orportions of code). A computer program can be deployed to be executed onone computer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio or video player, a game console, a GlobalPositioning System (GPS) receiver, to name just a few.

Computer readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto optical disks; andCD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,input from the user can be received in any form, including acoustic,speech, or tactile input.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described is this specification, or any combination of one ormore such back end, middleware, or front end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of any invention or of what may beclaimed, but rather as descriptions of features that may be specific toparticular embodiments of particular inventions. Certain features thatare described in this specification in the context of separateembodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable sub-combination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asub-combination or variation of a sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

1.-24. (canceled)
 25. A method of evaluating interacting RF devices,comprising: receiving selected interactivity parameters for a firstvirtual spectrum user from a first contestant; receiving selectedinteractivity parameters for a second virtual spectrum user from asecond contestant, wherein the first and second contestants are notcoordinated with each other and each contestant is not aware of theselected interactivity parameters from the other contestant; providing avirtual RF environment for the first and second spectrum users whereinthe selected interactivity parameters for the first virtual spectrumuser controls the operation of the first virtual spectrum user andaffects the performance of the second virtual spectrum user; evaluatingthe performance of the first and second virtual spectrum users in thevirtual RF environment wherein the step of evaluating the performance ofthe first virtual spectrum user includes evaluating a message throughputof the first and second virtual spectrum users in the virtual RFenvironment; assigning a score to the first virtual spectrum user as afunction of the evaluated performance of the first virtual spectrumuser; assigning a score to the second virtual spectrum user as afunction of the evaluated performance of the second virtual spectrumuser; and identifying whether the first or second virtual spectrum userwas assigned the higher score.
 26. The method of claim 25 wherein thestep of evaluating the second virtual spectral user includes evaluatinga message throughput of the first and second virtual spectrum users inthe virtual RF environment
 27. The method of claim 25 wherein thevirtual RF environment includes noise, propagation loss, multipathtransmissions and interferers.
 28. The method of claim 25 wherein theselected interactivity parameters are provided from a remote locationthrough a web browser during the step of evaluating.
 29. The method ofclaim 25 wherein the selectable interactivity parameters includefrequency, bandwidth, power and modulation.
 30. The method of claim 25wherein the selectable interactivity parameters include location, speed,direction of motion, and antenna parameters.
 31. The method of claim 30wherein the antenna parameters include type, elevation, gain, phasepolarization and orientation.
 32. The method of claim 25 wherein thefirst virtual spectrum user is a communication network.
 33. The methodof claim 25 wherein the second virtual spectrum user is a jammer. 34.The method of claim 25 wherein the second virtual spectrum user is aprimary user of the communication network having a selectable stationarylocation.
 35. The method of claim 33 wherein the primary user is atleast one of a radar, a navigation device, or a television transmitter.36. The method of claim 33 wherein the second virtual user is assignedpoints as a function of reducing the throughput of a communicationsnetwork in the virtual RF environment.
 37. A method of evaluatinginteracting RF devices, comprising: receiving selected interactivityparameters for a plurality of virtual spectrum users, wherein each ofthe plurality of virtual spectrum users are not coordinated with eachother and virtual spectrum user is not aware of the selectedinteractivity parameters from other virtual spectrum users; providing avirtual RF environment for the plurality of spectrum users, wherein theselected interactivity parameters for a first virtual spectrum usercontrols the operation of the first virtual spectrum user and affectsthe performance of the other virtual spectrum user; evaluating theperformance of a first virtual spectrum user in the virtual RFenvironment, wherein the step of evaluating the performance of the firstvirtual spectrum user includes evaluating the message throughput of allvirtual spectrum users in the virtual RF environment; assigning a scoreto the first virtual spectrum user as a function of the evaluatedperformance of the first virtual spectrum user; assigning a score to theother virtual spectrum users as a function of an evaluated performanceof the other virtual spectrum users; and identifying a virtual spectrumuser assigned the highest score among the plurality of virtual spectrumusers.
 38. The method of claim 37 wherein a second virtual user isassigned points as a function of detecting a jammer in the virtual RFenvironment.
 39. The method of claim 37 further including the step ofchanging the virtual RF environment in real-time and evaluating andassigning scores to the plurality of virtual spectrum users in thechanged virtual RF environment.
 40. The method of claim 37 furthercomprising the step of displaying the assigned scores of the pluralityof virtual spectrum users.
 41. The method of claim 37 further comprisingdisplaying the geographic location of the virtual RF environment and thelocation of the plurality of virtual spectrum users on a map.
 42. Asystem for evaluating interacting RF devices, comprising: a firstvirtual spectrum user having first selectable interactivity parameterscontrolled by a first contestant through selection of the firstselectable interactivity parameters; a second virtual spectrum userhaving second selectable interactivity parameters controlled by a secondcontestant through selection of the second selectable interactivityparameters; wherein the first and second contestants are not coordinatedwith each other and each contestant is not aware of the selectedinteractivity parameters from the other contestant; a real-time modelerprocessor responsive to the first and second virtual spectrum users inreal time to track the physical location of the first and second virtualspectrum users; a virtual spectrum processor responsive to the real-timemodeler processor to emulate a virtual RF environment in which the firstand second virtual spectrum users operate wherein the selectedinteractivity parameters for the first virtual spectrum user controlsthe operation of the first virtual spectrum user and affects theperformance of the second virtual spectrum user; a database of the firstand second selectable interactivity parameters for the first and secondvirtual spectrum users in communication with the first and secondcontestants through the respective first and second web browsers; anevaluation processor for evaluating the performance of the first andsecond virtual spectrum in the virtual RF environment and assigning ascore to the first and second virtual spectrum users as a function ofthe evaluation, wherein the evaluation of the performance of the firstvirtual spectrum user includes evaluating a message throughput of thefirst and second virtual spectrum users in the virtual RF environment.43. The system of claim 42 wherein the evaluation of the performance ofthe second virtual spectrum user includes evaluating a messagethroughput of the first and second virtual spectrum users in the virtualRF environment.
 44. The system of claim 42 wherein the first virtualspectrum user is a first communication network and the second virtualuser is a second communication network.
 45. The system of claim 42wherein the virtual RF environment includes noise, propagation loss,multipath transmissions and interferers.
 46. The system of claim 42wherein the first and second selected interactivity parameters includeselectable transmission parameters and selectable physicalcharacteristics controllable from a remote location through a webbrowser.
 47. The system of claim 46 wherein the selectable interactivityparameters include frequency, bandwidth, power and modulation.
 48. Thesystem of claim 46 wherein the selectable physical characteristicsinclude location, speed, direction of motion, and antenna parameters.49. The system of claim 48 wherein the antenna parameters include type,elevation, gain, phase polarization and orientation.